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which text data to be displayed is shown. Image progression information can 
be easily produced by selecting a picture to be shown in each scene from an 
event area, selecting a piece of music to be played from a song selection 
menu 22, with which a piece of music is selected and selecting a scenario to 
which effects and changing scenes are set in advance with a scenario 
selection menu button 24. 
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[Claim(s)] 

[Claim 1] The musical piece information corresponding to the musical piece 
which should be performed, and the image progress information that the 



event about the image reproduced corresponding to progress of the mustcal 
piece is specified, t. is edit equipment o, the mummedia information which can 
carry out creation edit of the sequence information which has the event 
information corresponding to the even, specified using said image progress 
information. By having the scenario information which recorded 

predetermined image progress setting-ou. information, and choosing and 
assigning said scenario information to the musica, piece data concerned 
Multimedia information edit equipment characterized by being made as 
I assign / an effect. - scene information, etc. corresponding to each scene , 

S'alm Z Said multimedia information edit equipment according to ctaim 1 
2 Tcterized by being made as [ substitute , said musica, piece information ,. 

The musica, piece information corresponding to * 
Lcn shou,d be performed, and the ,mage progress information ha, the 
even, abou, ,he image reproduced corresponding ,o progress o the « -,ca, 
pice is specified, „ is based on ,he sequence information which has .he 

progress in.orma.ion. Wi,h the performance of said musical piece 1s, means 
T. e muUimedia information regenerate apparatus which r^rc .duces 

image information corresponding to progress of .his musical p,ece, and ,o 

perform performance processing o, said musica, piece. The muU,med,a 
nfol. ion regenerate appara.us charaCerized by having 2nd means .o 

TZt image informafion based on said image progress informafion and 
St even. in,orma.ion, and 3rd means to disp.ay the image informal 

STS — Ze — ion corresponding .o ,e musica, piece 
lob should be performed «o a computer, and the image progress 



piece. Said pregrem The 1st function to make said computer perform 
performance processing of said musical piece. The record med.um which 
ecorded the muitimedia information piaybacK program charactered by be, 
It performs the 2nd function to make image information genera, based on 
said image progress information and said event information, and the 3rd 
Z In on which the image information generated by .aid 2nd funCon ,s 

- - — — whi * recorded me sequenoe ?:r n for 

reproducing image information with the performance of a mus,cal p.ece 

: : :ig - - — - r^rir 

The musicai piece information corresponding to the mus.cal p.ece wh,ch 
h i performed, and the image progress information that the even about 

said information arranged in order ot t.m. 9 „ h;irac terized by 

— r:r=-= — - 

each event, or said each event. 



[Detailed Description of the Invention 
performance of a musica, piece corresponding to progress of ed„ 



regenerative apparatus of the multimedia information which can reproduce 
image information corresponding to progress of this musical piece, the record 
medium which recorded the multimedia information playback program, and 
the musical piece performed. 
[0002] 

[Description of the Prior Art] Making image information and musical-sound 
information correspond, and reproducing them in a personal computer etc., 
conventionally, is known. There is software which creates the digital photo 
album of the original copy which combined the illustration, an alphabetic 
character, a sound, etc. on the personal computer as the example based on 
the photograph which the user itself took. A photograph is reproducible based 
on the photograph which the user itself took according to such software in 
order like a slide on a personal computer combining an illustration, an 
alphabetic character, a sound, etc. And when desired BGM can be added to 
desired 1 or two or more desired photographs and the page of an album is 
turned over, the screen effectiveness, such as wipe, fade-in, and fade-out, is 
added, and it is made as [ give / the screen effectiveness, such as performing 
page turning over, ]. 
[0003] 

[Problem(s) to be Solved by the Invention] In the conventional technique 
mentioned above, playback of an image is main and was not able to create 
easily the sequence information on image progress that progress of a musical 
piece like a video clip was met. Moreover, in the software mentioned above, 
the screen effectiveness at the time of the switch of the background of the 
display screen or a photograph by which two or more preparation was carried 
out (fade-in, fade-out) was put in order suitably, and image progress 
information was created. Therefore, there was what has difficult for a beginner 
creating image progress information so that a series of photographs or the 
flow of effectiveness may become effective. Furthermore, at the time of edit of 
image progress information, editing tasks, such as substitution and elimination, 
needed to be performed for every [ to edit ] photograph and every 
effectiveness, and the editing task became troublesome. 
[0004] Then, this invention loses such inconvenience, makes progress of a 



musical piece Maine, and aims at offering the edit and the regenerative 
apparatus of multimedia information which can create easily the sequence 
information on image progress that progress of a musical piece was met, and 
the record medium which recorded the multimedia information playback 
program. Moreover, it aims at offering the edit and the regenerative apparatus 
of multimedia information into which the image information and effectiveness 
which are reproduced corresponding to progress of the musical piece 
concerned can be edited easily, and the record medium which recorded the 
multimedia information playback program. Furthermore, it aims at offering the 
record medium with which the sequence information which has the new DS 
which can reproduce an image in the format in alignment with progress of the 
musical piece performed was recorded. 
[0005] 

[Means for Solving the Problem] In order to attain the above-mentioned object, 
the multimedia information edit equipment of this invention The musical piece 
information corresponding to the musical piece which should be performed, 
and the image progress information that the event about the image 
reproduced corresponding to progress of the musical piece is specified, It is 
edit equipment of the multimedia information which can carry out creation edit 
of the sequence information which has the event information corresponding to 
the event specified using said image progress information. It has the scenario 
information which recorded predetermined image progress setting-out 
information, and is made by choosing and assigning said scenario information 
to the musical piece data concerned as [ assign / an effect, ** scene 
information, etc. corresponding to each scene / collectively ]. Moreover, it is 
made as [ substitute / said musical piece information ]. 

[0006] Furthermore, the multimedia information regenerative apparatus of this 
invention The musical piece information corresponding to the musical piece 
which should be performed, and the image progress information that the 
event about the image reproduced corresponding to progress of the musical 
piece is specified, It is based on the sequence information which has the 
event information corresponding to the event specified using said image 
progress information. With the performance of said musical piece 1st means 



to be the multimedia information regenerative apparatus which reproduces 
image information corresponding to progress of this musical piece, and to 
perform performance processing of said musical piece, It has 2nd means to 
generate image information based on said image progress information and 
said event information, and 3rd means to display the image information 
generated by said 2nd means. 

[0007] The record medium which recorded the multimedia information 
playback program of this invention further again The musical piece 
information corresponding to the musical piece which should be performed to 
a computer, and the image progress information that the event about the 
image reproduced corresponding to progress of the musical piece is specified, 
It is based on the sequence information which has the event information 
corresponding to the event specified using said image progress information. 
With the performance of said musical piece It is the record medium which 
recorded the program for reproducing image information corresponding to 
progress of this musical piece. Said program The 1st function to make said 
computer perform performance processing of said musical piece, the 2nd 
function to make image information generate based on said image progress 
information and said event information, and the 3rd function on which the 
image information generated by said 2nd function is displayed are performed. 
[0008] The record medium which recorded the sequence information on this 
invention further again It is the record medium which recorded the sequence 
information for reproducing image information with the performance of a 
musical piece corresponding to progress of this musical piece. Said sequence 
information The musical piece information corresponding to the musical piece 
which should be performed, and the image progress information that the 
event about the image reproduced corresponding to progress of the musical 
piece is specified, It has the event information corresponding to the event 
specified using this image progress information. Said image progress 
information The event related information for specifying the event timing data 
which specify the timing which the event about said image generates, and the 
data corresponding to the event concerned contained in said event 
information is said information arranged in order of timing. Said event 



information It considers as the data which specify the data used for the data 

used for said each event, or said each event. 

[0009] 

[Embodiment of the Invention] The sequence information for reproducing 
image information with reference to a drawing hereafter corresponding to 
progress of multimedia information edit and the regenerative apparatus of this 
invention, the record medium which recorded the multimedia information 
playback program, and the musical piece performed is explained to a detail. In 
addition, multimedia information edit and the regenerative apparatus of this 
invention are explained here as what is realized by the general-purpose 
personal computer, the multimedia information edit which operates on it, and 
the playback program, although all of realizing using the edit and the playback 
program of multimedia information which operate realizing as a processor of 
dedication or a general-purpose personal computer, and on it are possible. 
[0010] Drawing 1 is the block diagram showing an example of edit of the 
multimedia information on this invention, and the hardware configuration of 
the gestalt of 1 operation of a regenerative apparatus. In this drawing, ROM 
which memorizes CPU to which 1 performs edit of this multimedia information 
and control of the whole regenerative apparatus, the control program with 
which CPU1 performs 2, and various kinds of control information, RAM for 
which 3 is used as a program, various control information, various kinds of 
data storages mentioned later, and temporary storage (buffer), and 4 are a 
timer which directs timing, such as timer-interruption processing, to said CPU1 
while directing time of day. In addition, you may make it memorize the edit 
and the playback program of multimedia information concerning this invention 
to any of said ROM2 or said RAM3. It will store in said RAM3, once acquiring 
the program concerned from the server computer connected to the 
communication network 17 through the communication link interface 16 which 
reads the program concerned from the external storage 1 1 mentioned later, 
and stores in RAM3, or is mentioned later to a hard disk drive unit etc., when 
memorizing to RAM 3. 

[001 1] Moreover, it is a display circuit for a detector for a detector for a 
keyboard for 5 to input text data, function data, etc. and 6 to connect this 



keyboard 5 and 7 to connect pointing devices, such as a mouse, and for 8 
connect this pointing device 7 and 9 to control a display unit, and for 10 
control the display of the image to this display unit 9. Furthermore, 1 1 is 
external storage and various external storage, such as a hard disk drive unit, 
CD-ROM, and FD, MO, DVD, ZIP, jaz, is connected. Moreover, an 
input/output interface circuit for 12 to connect with sound-source equipment 
14 and 13 are the sound systems by which a MIDI device and 14 were 
connected to sound-source equipment, and 15 was connected to said sound- 
source equipment 14. Furthermore, a communication link interface circuitry for 
16 to connect with a communication network 17 and 17 are communication 
networks, such as the telephone line, the Internet, and LAN. 
[0012] Before explaining multimedia information edit and the playback 
program of this invention, various kinds of data held at said RAM3 are first 
explained with reference to drawing 2 at the time of multimedia information 
edit and playback program execution, (a) of drawing 2 is effect information, 
and the image (animation) information used as background effectiveness at 
the time of image reconstruction is stored in the usual animation format of AVI 
(Audio-Video Interleave) etc. Various kinds of effect information, such as an 
effect to which there are an effect for pictures and an effect for texts in this 
effect information, for example, snow seems to fall, is beforehand stored in 
the predetermined field in said RAM3 two or more kinds. 
[0013] (b) of drawing 2 is ** scene information, and in order to connect two 
scenes effectively, it is animation information used in the bond location of a 
scene. This ** scene information is the information for giving effectiveness, 
such as making it move to the following scene etc., as fade-in is carried out to 
a back scene from a front scene or the page of a book is turned over. Two or 
more kinds also of this ** scene information are beforehand stored in the 
predetermined field in said RAM3. 

[0014] (c) of drawing 2 is picture information, and is the image information of 
Maine in each scene. Any of an animation and a still picture are sufficient as 
this picture information, it is a general-purpose format of GIF, JPEG, AVI, etc., 
and is beforehand stored in the predetermined field in said RAM3 two or more 
kinds. Moreover, a user can also do additional record of the new picture 



information to the predetermined field concerned. 

[0015] (d) of drawing 2 is musical piece data, for example, is musical piece 
data created based on SMF (Standard MIDI File). A user is able to store two 
or more kinds also of this musical piece data in the predetermined field in said 
RAM3 beforehand, and to do additional record at arbitration. 
[0016] (e) of drawing 2 is scenario information, and this scenario information 
is information which sets up progress of an image. One scenario information 
is constituted by three kinds of information, a scenario number, title 
information, and progress information, so that it may illustrate. A scenario 
number is an identification number given for every class of scenario, and the 
image progress information concerning [ title information ] the title image at 
the time of playback initiation of an image is included. Moreover, progress 
information consists of information showing the appearance pattern and 
scene break location of the effect for the effect for pictures, ** scene 
information, a text, and texts, and sequential storing of the image progress 
information for every scene is carried out. In addition, the data which direct 
time amount are not contained in this progress information. 
[0017] As for said scenario information, the thing of two or more classes, such 
as video clips (pop, techno, etc.) and movies (8mm, theater, etc.), is 
beforehand prepared into said RAM3. For example, if the scenario information 
for 8mm movies is taken for an example, the image progress information that 
a number is displayed on "3", "2", "1", and order is included in the title 
information, and it is made as [ add / the effect with which the image progress 
information for every scene was also similar to actual 8mm movie ]. Thus, a 
user becomes possible [ creating easily the effective image progress 
information based on the scenario information which only chooses scenario 
information and was this chosen ] by preparing typical scenario information 
beforehand. For example, a user is choosing the musical piece performed so 
that it may mention later, and choosing said scenario information, after 
determining image data to display in division of a scene, and each scene, and 
becomes possible [ setting up automatically said title screen, an effect 
effective for each scene, and ** scene information ]. 
[0018] Next, drawing 3 is referred to and the sequence information for 



reproducing the multimedia information created or edited using multimedia 
information edit and the playback program of this invention is explained. So 
that (a) of drawing 3 may be drawing showing the one whole sequence 
information structure and it may illustrate each sequence information The 
sequence number for identifying this sequence information, the musical piece 
information corresponding to the musical piece used for this sequence 
information, The information about the various image events generated along 
with the performance of said musical piece consists of end data in which the 
image progress information stored in order of generating timing, the event 
information about said various image events, and termination of this sequence 
information are shown. In addition, in the gestalt of this operation, although 
the sequence number is used as information for identifying said sequence 
information, other information, such as a name of the sequence information 
concerned, may be used. Moreover, as said musical piece information 
corresponding to the musical piece concerned, the musical piece tbe data for 
specifying the musical piece data (musical piece sequence data) concerned 
and the musical piece data itself [ concerned ] can be used. In addition, with 
the gestalt of this operation, the musical piece tbe data which specify musical 
piece data shall be used. 

[0019] (b) of drawing 3 is drawing showing the content of said image progress 
information. As shown in this drawing, the chunk size data which image 
progress information shows the magnitude of the chunk type data in which it 
is shown that this information block is image progress information, and this 
image progress information block are stored in that head part. The information 
relevant to the event timing data which specify the timing to generate about 
each image event hereafter, and the image event concerned, That is, let 
various image event related information (various related information), such as 
picture related information, effect related information, text related information, 
and ** scene related information, be event sequence data by which sequential 
storing was carried out at the order of generating timing. Here, in this 
invention, sake [ progress of a musical piece / basic ], said event timing data 
are made into the timing information corresponding to progress of a musical 
piece, for example, the data which made the MIDI clock the unit. Moreover, 



performance termination directions data are data for directing to terminate the 
performance of musical-sound information at the time of termination of image 
display. 

[0020] Image event related information (various related information), such as 
said picture related information, effect related information, text related 
information, and ** scene related information So that all may be made into the 
data which have the same structure and it may illustrate The class data in 
which the class (a picture, an effect, a text, or ** scene) of the related 
information is shown, The display time data which shows the time amount 
length of an applicable scene, the time resolution data in which the playback 
time resolution of the image information to reproduce is shown, The setting- 
out information about setting out of the display size of each image or a text, a 
display position, a font, etc., etc., It consists of event number data for 
specifying the corresponding event information included in the event class 
information which shows any of a still picture, an animation, or a text the 
image event is, and the event information on (a) of drawing 3 mentioned 
above. 

[0021] (c) of drawing 3 is drawing showing the content of the event 
information which follows said image progress information. Event information 
is the record section of the image information corresponding to the event 
number stored in [ said various ] related information. As shown in this drawing, 
the event tbe data to which the chunk type which event information shows that 
this information block is event information, and the chunk size the magnitude 
of this event information block is indicated to be are stored in that head part, 
and specify a corresponding event as the event number of each [ the following 
and ] event are stored. Here, when the event concerned is a text, the text data 
itself is recorded as event tbe data, and when an event is a still picture or an 
animation, the data (event tbe data) in which the record section where the 
image data is recorded is shown are recorded. 

[0022] Since event timing information which shows the generating timing of an 
image event is made into the timing information corresponding to progress of 
a musical piece according to the sequence information on this invention that it 
has the above DS, image information can be reproduced corresponding to 



progress of a musical piece performance. Moreover, since event directions 
data are not directly written in in image progress information, rewriting of the 
data of the record format using event directions data and the record (image 
information was used) format of not using event directions data can carry out 
easily only by rewriting said event information field. 
[0023] Drawing 4 is drawing showing an example of the initial screen 
displayed on said display unit 9, when it performs on the personal computer 
which showed multimedia information edit and the playback program of this 
invention to said drawing 1 . In this drawing, 21 is a sequence information 
(Sequence Info) viewing area. In this sequence information-display field 21, 
the direction of an axis of abscissa is a time-axis, for example, the graduation 
corresponding to the vibrant tune location of a musical piece is attached. And 
three fields, the song field (SONG) which displays musical piece data, the 
picture field (PICTURE) which displays image data, and the text area (TEXT) 
which displays text data, arrange, and are prepared. Thus, this sequence 
information-display field 21 is a field which displays collectively the multimedia 
information with which the musical piece data, image information, and text 
information which are specified using sequence information, i.e., musical 
piece information, were combined on the same time-axis, and a user will 
perform creation of new sequence information, and already created edit of 
sequence information on this sequence information-display field 21. 
[0024] 22 is a menu button for the musical piece selection for choosing a 
musical piece performing, and can choose the musical piece data beforehand 
memorized as mentioned above by the pull down menu method by choosing 
this menu button 22 with a mouse etc. 23 is a menu button for sequence 
selection, and is for choosing the sequence information already created. 
Furthermore, 24 is a menu button for the scenario selection for choosing said 
scenario information which is created beforehand and prepared. 
[0025] 25 is a play field and is a field which displays the image information 
when reproducing the sequence information by which it is indicated by current 
on said sequence information-display field 21. 26 is an event field, and when 
editing sequence information, it is a field for choosing information, such as a 
selectable picture, an effect, and a ** scene. When this event field 26 is 



constituted by the menu of for example, a tab dialog type, the tab indicated to 
be a picture, an effect, and a ** scene is formed and each tab is chosen so 
that it may illustrate, the icon of the selectable thing of that selected event is 
displayed. In drawing, the case where a picture is chosen is shown and the 
icon which shows the selectable picture of 1-7 is displayed. A user can assign 
a desired picture to the scene by choosing the icon of the picture which you 
want to display on the scene of the request of the sequences currently edited 
or created, and dragging to the picture field concerned. 
[0026] 27-30 are the control carbon buttons for controlling playback of the 
sequence concerned, and, as for playback according [ accord / 27 / rewinding 
/ 28 ] to a rate, and 29, a halt and 30 are usually the carbon buttons for a rapid 
traverse. A user can control playback of the sequence concerned by clicking 
these control carbon buttons 27-30 with pointing devices, such as a mouse. 
[0027] In multimedia information edit and the playback program of this 
invention, drawing 5 is drawing for explaining the appearance of said display 
screen when sequence information is created, when sequence information is 
read. As shown in this drawing, selected musical piece data are displayed on 
the song field in said sequence field 21. Here, by double-clicking choosing this 
musical piece data and choosing a predetermined menu or this musical piece 
data, the edit window of the musical piece data concerned is opened, and edit 
of musical piece data is attained like the case where it is the usual sequencer 
software. 

[0028] Moreover, a picture field and a text area are divided into each scene by 
the scene break line shown by the vertical line all over drawing, and the 
picture information and text information corresponding to each scene are 
displayed. In the illustrated example, the text information in a title screen 
serves as an identifier of "a hike" and star who are the name of the sequence 
information concerned, and the author. Moreover, the icon of the picture 
chosen as the picture field of the 1st scene by the user shown by "1", the icon 
which shows a predetermined effect to the bottom of it, and the icon which 
shows a ** scene further predetermined to the bottom of it are displayed. 
Moreover, the icon which directs the caption for explaining the date and its 
scene and the effect (how change of a foreground color and a display carry 



out [ For example, ]) to a text is displayed on the text area. Furthermore, the 
picture shown by "2" is chosen as the picture field of the 2nd scene. In 
addition, the effect and ** scene information over a picture are not set to this 
2nd scene, and the effect of a text and a text is not set to it, either. Only the 
effect of a background is set up, for example and the icon to the picture and it 
which are shown by "3" which shows that is displayed on the 3rd scene further 
again. Hereafter, the icon which the picture similarly corresponding to [ to a 
scene ] each scene in a picture field of the 4th and 5th - which follows, a 
setting-out effect, and a ****** scene are displayed by the icon, and shows the 
effect to the text information displayed on each scene in a text area and its 
text is displayed. 

[0029] As shown in this drawing, while displaying the sequence information 
under creation or edit, when a user uses the menu button 22 for said musical 
piece selection and chooses another musical piece, the image sequence to 
display can change only the musical piece made to perform as it is. Moreover, 
the image information displayed in the musical piece and each scene which 
should be performed can make a similar change of an effect and the ** scene 
as it is by choosing a new scenario using the menu button 24 for said 
scenario selection. 

[0030] Moreover, a user can create the sequence information on desired by 
dragging a desired thing to the field of the scene of a request of said 
sequence information field out of the icon of various kinds of pictures 
displayed on said event field, an effect, and a ** scene. Moreover, when 
editing the sequence information already created, a picture, effect information, 
etc. which are displayed on said sequence information field can be chosen, 
and edit of sequence information can be performed by deleting or copying 
using the Edit menu which is not illustrated. Moreover, it gets down to 
operating said control carbon buttons 27-30, and the sequence information 
under creation or edit is reproduced suitably, and it can work, checking a work 
content. 

[0031] Next, the operation flow chart of multimedia information edit of the 
gestalt of this operation and a regenerative apparatus is explained. Drawing 6 
is the flow chart of the Maine processing in this multimedia information edit 



and regenerative apparatus. In addition, this multimedia information edit and 
regenerative apparatus are constituted so that it may operate by the Maine 
processing shown in drawing 6 , and three kinds of interrupt processing 
mentioned later, actuation which reproduces the sequence information 
created or edited is performed by said three kinds of interrupt processing - ** 
— it becomes. 

[0032] If activation of the Maine processing of drawing 6 is started, first, the 
initialization process of step S1 will be performed and the display process of 
said initial screen and the initialization process of various flags or various data 
areas will be performed. Next, it progresses to step S2 and it is judged 
whether a run (RUN) flag is "1 ." Here, a RUN flag is a flag which shows 
whether playback of the musical piece and image based on sequence 
information is performed, and when playback is performed and "1" and 
playback are not performed, it is set to "0." Here, when a RUN flag is "0", edit 
processing of step S3 is performed. In this edit processing, a user can 
perform creation or edit of sequence information. About the detail of this edit 
processing, it mentions later with reference to drawing 7 . 
[0033] Moreover, regeneration of the musical piece and image based on 
sequence information is performed, when a RUN flag is "I", or when said edit 
processing is completed, it progresses to step S4, and it is judged whether 
playback initiation was directed. Initiation of this playback is performed by 
operating the directions from the menu which is not illustrated, or said 
playback carbon button 28. When playback initiation is directed, it progresses 
to step S5 and a playback initialization process is performed. In this playback 
initialization process, initialization processing of the flag used for playback of 
the musical piece data concerned and image information, a register, a buffer 
area, etc., etc. is performed, moreover, the result of reading appearance of 
the progress information for the specified quantity having been carried out 
from the head of the image progress information within the sequence 
information reproduced, and having interpreted this progress information that 
carried out reading appearance is recorded on a record section temporarily 
which was secured into said RAM3. And it progresses to step S6, said RUN 
flag is set to "1", and the read-out location in a record section is set 



temporarily [ said ] for image reconstruction. 

[0034] When the judgment result of said step S4 is NO after processing of 
said step S6 is completed or, it progresses to step S7 and judges whether 
playback termination was directed. This playback termination is directed by 
actuation of the playback termination directions by the menu which is not 
illustrated, or said playback termination carbon button 29. And when the result 
of this judgment is YES, while progressing to step S8 and setting a RUN flag 
to "0", silence processing of musical sound and the post process of image 
display are performed. 

[0035] After said step S8 is completed when the judgment result of said step 
S7 is NO or, it progresses to step S9 and other processings, such as 
processing corresponding to actuation of said fast forward button 30 or said 
rewind button 27, are performed. This rapid traverse or rewinding processing 
is realizable by performing regeneration to the forward direction or hard flow 
for said every scene. And next, it progresses to step S10 and judges whether 
this multimedia information edit and termination of regeneration were directed, 
and when this judgment result is YES, this Maine processing is ended. 
Moreover, when the judgment result of step S10 is NO, return, said step S2 - 
step S10 are repeated and performed to said step S2. 

[0036] Drawing 7 is a flow chart which shows the actuation in edit processing 
of said step S3, if this edit processing is started, in step S11, it is new creation 
of sequence information first - or it is judged whether it is edit of the existing 
sequence information. Here, when a user chooses a musical piece newly by 
the menu button 22 grade of said drawing 4 , it is judged with it being new 
creation, and progresses to step S12, and selection of musical piece data is 
performed. Thus, since the sequence information on this invention is created 
by making musical piece data correspond and assigning the information about 
an image, when creating sequence information newly, in step S12, selection 
of musical piece data is performed first. 

[0037] On the other hand, when the sequence information for which the user 
is already created by the menu button 23 grade for sequence selection of said 
drawing 4 is chosen, at the time of the edit of the existing sequence 
information instead of new creation, it progresses to step S13, and reading 



appearance of the sequence Information set as the obiect of edit . chosen 
and earned out. After step S12 or S13 is completed, it progresses to stop ,314 
and display processing is performed. When said step S12 is performed, the 
musical piece data chosen as said song field in the inifia, screen shown in 
said drawing are displayed and step S13 is performed by this, the d.spiay 
screen shown in said drawing 5 will be displayed. 
,0038, And it progresses to step S15 and it is judged whather selaCon 
actuation o, a scenario was performed. Section actuation of .his scenano s 
perforated by actuation o, the menu button 24 for said scenario safcohoMhe 
scenario information (said drawing (.)) beforehand stored ,n sa,d * 
choosing and clicking this menu button 24 using said pointing dev,ce ,s 
expressed for example, as a pu„ down menu method, and „ performs by 
choosing the scenario information on desired. 

[0039, when scenario selection actuation is performed, i. progresses to step 
S 6 and ,s a thing for substitution directions o, selection o, said scenano of a 
scenario - or it is judged whether it is scenario selection ao.ua.on ^ he 
creation time of new sequence information. When the scenano 
U p ,o sequence information, it judges that this scenario se*c , on , s 
substitution directions of a scenario, and it progresses to step S17 A. step 
S17 . the image progress information within the sequence ' 
current edit is rewritten based on the selected scenano ,nformat,on. That ,s 
Z2. .he effect, and - scene related in.orma.ion within .he image p = 
information concerned bundle up to the information equ,va,en, to the setected 
scenario information, and are rewritten. In addition, .ha picture arrange aUhe 
musical piece da. and each scene which are chosen 
«me Moreover, rewriting is no. performed when an effect and scene 
Imafion are not contained in image progress information aUhe me 
scenario subs.i.u.ion. Thus, modification o. a title, an effect, and scene 

L information is attained coileCively. Per example by change set- 
U p scenario information into jazz from pop, -respon ,ng o ,, .hey re 
col.ect.vejy changed by «. informa,ion, various effects, and scene 
information, and i, becomes possible .o display .ha image of a completely 
different ambient atmosphere. 



[0040] On the other hand, when scenario information, such as new sequence 
information creation time, is not set up and said scenario selection is 
performed, the judgment result of said step S16 serves as NO, and step S18 
is performed. At this step S18, image progress information is created in 
sequence information in preparation based on the selected scenario 
information. That is, based on the progress information included in the 
selected scenario information, the image progress information concerning a 
title, an effect, and ** scene related information is created automatically. In 
addition, when the picture is already arranged at this time, arrangement of this 
picture is maintained as it is. 

[0041] After said step S17 or step S18 is performed when the judgment result 
of said step S15 is NO or, step S19 is performed. It is judged at step S19 
whether substitution of a musical piece was performed. Substitution of this 
musical piece is directed by operating said menu button 22 for musical piece 
selection with a pointing device. When a musical piece which said menu 
button 22 for musical piece selection is operated, and is different from the 
musical piece by which current selection is made is chosen, the judgment 
result of this step S19 serves as YES, and rewriting of the musical piece 
information within sequence information is performed in step S20. In addition, 
other information within sequence information is not changed at all at this time. 
It enables this to change only the musical piece information to perform simply. 
[0042] When the judgment result of said step S19 is NO after processing of 
said step S20 is completed or, it progresses to step S21 and it is judged 
whether informational record was directed. Directions of record of this 
information are processing which judges whether edit processing of various 
information was performed by the user on the display screen shown in said 
drawing 5 . By the user A desired picture and the icon of an effect or a ** 
scene are chosen from a certain edit processing 26, for example, said event 
field, to said sequence information field. When processing of dragging to said 
picture field or text area is performed, this judgment result serves as YES. 
And at this time, it progresses to step S22 and sequence information [ / in the 
sequence information-display field 21 ] is recorded. 

[0043] When the judgment result of said step S21 is NO after processing of 



said step S22 is completed or, it progresses to step S23 and other edit 
processings are performed. As edit processing of these others, they are trial 
regeneration of a scene or a ** scene, entry processing of the text to a text 
area, etc., for example. For example, if a user double-clicks the text area of a 
desired scene, a text input screen is displayed and a user can enter desired 
text data in the text area. Moreover, preservation processing of the sequence 
information edited or created etc. is performed. Easily, it can create as 
mentioned above or a user can edit sequence information. 
[0044] Next, regeneration of the sequence information created or edited as 
mentioned above is explained. As mentioned above, regeneration of this 
sequence information is performed by three kinds of interrupt processing, (a) 
of drawing 8 is a flow chart which shows actuation of the 1st interrupt 
processing (interrupt processing 1). This interrupt processing 1 is processing 
which reproduces the musical piece data specified using the musical piece 
information within the sequence information currently reproduced, for example, 
is made as [ perform / for every clock timing of the clock pulse generated 24 
times between quarter note length ]. If said clock pulse is generated, this 
interrupt processing 1 (performance interrupt processing) will be started, and 
it is judged first whether the RUN flag is set to "1" in step S31 . And this 
interrupt processing 1 is ended, without performing anything, since it is in the 
condition that sequence information is not reproduced and of not performing 
when a RUN flag is "0." 

[0045] Moreover, when the RUN flag is set to "1", it progresses to step S32 
and judges whether with reference to said musical piece data ( drawing 2 (d)), 
it became the timing which should perform the event about the MIDI data 
contained in the musical piece data concerned. When having not reached 
event timing as a result of this judgment, this interrupt processing 1 is ended. 
Moreover, when having reached event timing, it progresses to step S33 and 
the musical piece event processing is performed. This processing is the same 
as the processing in the usual musical-sound generator, for example, is 
processing which generates the sound-source control parameter 
corresponding to the MIDI event concerned, and is outputted to said sound 
source 13. 



[0046] (b) of drawing 8 predicts the image progress information included in 
said sequence information currently reproduced, and is the flow chart of the 
2nd interrupt processing (interrupt processing 2) which creates the image data 
which should be displayed. This interrupt processing 2 (read-ahead interrupt 
processing) is performed 72 times between time intervals shorter than said 
interrupt processing 1, for example, quarter note length. If this interrupt 
processing 2 is started, first, it is judged in step S41 whether a RUN flag is "1", 
and when a RUN flag is "0", this interrupt processing 2 will be ended as it is. 
Moreover, when a RUN flag is "1", it progresses to step S42 and it is judged 
whether read-ahead processing is possible. For example, when read-ahead 
processing cannot be performed for the reasons of there being no free area in 
the field currently assigned as said temporary storage, this judgment result 
serves as NO and this interrupt processing 2 is ended. 
[0047] Moreover, when the judgment result of said step S42 is YES, the 
image progress information in the sequence information reproduced by 
progressing to step S43 is predicted by one scene. As said drawing 3 was 
explained, in image progress information The image event related information 
corresponding to the event timing data and it which show the generating 
timing of each image event, Namely, picture related information, effect related 
information, text related information, ** scene related information becomes a 
group, and is stored, and reading appearance of the image event related 
information which has the same event timing data is carried out as image 
progress information for one scene. Based on the event number contained in 
the image related information for this 1 predicted scene, the event data which 
correspond based on event tbe data are also read with reference to the event 
information in said sequence information. 

[0048] next, after thawing data, such as picture data which progressed and 
carried out this reading appearance to step S44 and by which the information 
compression was carried out, with reference to the display time data in said 
image related information, time resolution data, and setting-out information, 
the synthetic image information for every time resolution to which each image 
corresponding to the image information for every time resolution equivalent to 
the display time length concerned, i.e., the predicted picture for one scene, an 
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effect, a text, and a ** scene corresponds is created. And the information 
which progressed to continuing step S45 and was this created is recorded on 
said temporary storage in order of an event. Here, when image information is 
not recorded on the record section which said event tbe data direct, it treats 
as what the event tbe data did not begin and cut. For example, when the 
effect and the picture are directed and the information on an effect is not 
recorded, the information only based on a picture is created. Or you may 
make it use the specified image information and similar image information. 
Moreover, based on the event timing data contained in said image progress 
information, and the time resolution data in said image related information, the 
hour entry about the timing which reproduces said compound image 
information is created, and it stores in said primary store. That is, the time 
data which shows the interrupt timing of the interrupt processing 3 later 
mentioned for every scene, and the image reconstruction sequence data in 
the format of timing data + event (synthetic image) data after it will be 
memorized in said temporary storage. 

[0049] (c) of drawing 8 is a flow chart which shows actuation of the 3rd 
interrupt processing (interrupt processing 3) reproduced by reading the image 
information (image reconstruction sequence data) in which creation record 
was carried out by said interrupt processing 2 into the momentary record 
section in RAM3. This interrupt processing 3 (image interrupt processing) is 
performed the period based on the time resolution set up for said image 
progress information. Thereby, according to whether the image reproduced is 
a still picture or it is an animation, an execution cycle is changed and it 
becomes possible to attain speeding up of processing of operation. 
[0050] If the timer interruption of a period based on said time resolution 
concerned occurs, this interrupt processing 3 will be started and it will be first 
judged in step S51 whether said RUN flag is "1." And when a RUN flag is "0", 
this interrupt processing 3 is ended. Moreover, when a RUN flag is "1", it 
progresses to step S52 and data are read from said primary store. And it 
judges whether this read data is said time data (step S53), and when it is time 
data, it progresses to step S54, and this time data is set up in order to make 
said timer 4 generate this 3rd interrupt. By this, the interrupt timing of interrupt 



processing 3 will be changed based on the time data memorized in the 
change location of a scene. And data are again read from said primary store. 
On the other hand, when the judgment result of said step S54 is NO, it 
progresses to step S55 and judges whether the event (synthetic image 
information which should be displayed) corresponding to this interruption 
timing is recorded in the record section temporarily. And when there is no 
corresponding event, this interrupt processing 3 is ended. Moreover, when 
there is a corresponding event, it progresses to step S56, the image 
information concerned memorized in said temporary storage is outputted to a 
display circuit, and image event processing is performed. And it progresses to 
step S57, information [ finishing / processing ] is deleted from said temporary 
storage, and this interrupt processing 3 is ended. In addition, when the event 
in said step S55 is performance termination directions data, while setting a 
RUN flag to "0", in step S56, silence processing and the display post process 
of a musical-sound signal will be performed. 

[0051] In addition, in ****, although interrupt timing of said interrupt processing 
2 (read-ahead interrupt processing) was made into the timing relevant to a 
MIDI clock, it is not restricted to this and this interrupt timing can be made into 
the timing of arbitration. Moreover, before performing said read-ahead 
interrupt processing about the whole image progress information concerned at 
the time of sequence information playback initiation directions or opening of 
sequence information and starting playback actuation, you may make it store 
all synthetic image data (image reconstruction sequence data) in temporary 
storage. Thereby, it becomes reproducible [ stable ] even if it is a case so that 
the throughput of CPU may be insufficient. Moreover, in the same interrupt 
processing, it may be made to perform the above-mentioned interrupt 
processing 1 and the above-mentioned interrupt processing 3 by making the 
above-mentioned image event processing period (processing period of 
interrupt processing 3) into a musical piece event processing period (MIDI 
clock timing). 

[0052] Furthermore, you may enable it to assign two or more pictures and 
effects in the gestalt of the above-mentioned implementation, although one 
picture and an effect were assigned in 1 scene as shown in said drawing 5 , 



without being restricted to this. Although said effect, ** scene information, and 
picture information were made into the thing of the compressed format in **** 
further again, it is not restricted to this. Moreover, although [ said musical 
piece information ] it is MIDI data, it is not restricted to this and an audio signal 
may be used. 

[0053] Although musical piece information and image progress information 
were separately stored in sequence information in the gestalt of said 
operation, it is good further again also as a record format in which both who 
used musical piece information as musical piece event data, and have 
arranged musical piece event data and said image progress information 
(image event data) to time order were intermingled. Although the event 
directions data which direct the record section of each image information shall 
be recorded in said event information further again (said drawing 3 (c)), you 
may make it store image information directly like text information in event 
information. In the case of the gestalt of operation mentioned above, although 
the amount of data of sequence information can be lessened, the amount of 
data increases in this case. However, also in the computer by which the 
image data directed by event directions data is not stored, it becomes 
possible to reproduce the sequence information concerned. 
[0054] You may make it record said scenario information in said sequence 
information further again. In this case, it becomes possible by making the old 
scenario and a new scenario correspond at the time of scenario modification 
to make it serve as information for creating new image progress information. 
You may make it make said scenario information equivalent to musical piece 
information further again. For example, two or more scenario information is 
given for every musical piece, and it becomes possible to create easily the 
thing which gets the setting-out information within the scenario information 
concerned used to a response musical piece, then the more effective image 
progress information for every musical piece. You may make it include the 
break positional information of a scene in said setting-out information further 
again. Although substitution of a scenario or a musical piece is put in block or 
the whole and it is made to perform it in said edit processing further again, it i 
not restricted to this and you may enable it to perform substitution only about 



the predetermined range. 

[0055] Although a general-purpose personal computer shall be used in the 
gestalt of operation mentioned above, it may be made to perform processing 
mentioned above with two or more equipments connected using means of 
communications, such as electrohone having sound-source equipment and 
automatic performance equipment, or MIDI, various networks, further again. 
[0056] 

[Effect of the Invention] As explained above, according to multimedia 
information edit and the regenerative apparatus of this invention, it becomes 
possible to create and record the sequence information on image progress in 
the format in alignment with the flow of ******, and the digital album of the 
video clip style etc. can create easily. Moreover, image progress information 
with an effective effect and flow can be created easily. Furthermore, the 
ambient atmosphere and the musical piece information matched on the image 
progress concerned can be changed, maintaining the information about 
progress of the image created now by the easy editing task. According to the 
record medium which recorded the sequence information on this invention, it 
becomes possible to reproduce image information corresponding to playback 
of a musical-sound signal further again. 



DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing 1] It is the block diagram showing an example of multimedia 
information edit of this invention, and the hardware configuration of a 
regenerative apparatus. 

[Drawing 2] It is drawing for explaining the various data memorized in RAM3. 
fDrawing 3] It is drawing for explaining the sequence information on this 
invention. 

fDrawing 4] It is drawing showing an example of the initial screen of 



multimedia information edit of this invention, and a regenerative apparatus. 
[Drawing 5] It is drawing showing an example of the screen at the time of 
multimedia information edit of this invention, and the actuation in a 
regenerative apparatus. 

[Drawing 6] It is a flow chart for explaining multimedia information edit of this 
invention, and the Maine processing in a regenerative apparatus. 
[Drawing 7] It is a flow chart for explaining edit processing of multimedia 
information. 

[Drawing 8] It is a flow chart for explaining regeneration of the multimedia 
information in multimedia information edit and the regenerative apparatus of 
this invention. 
[Description of Notations] 

1 CPU, 2 ROM, 3 RAM, 4 Timer, 5 Keyboard, 6, eight detectors, 7 A pointing 
device, 9 Display unit, 10 A display circuit, 11 External storage, 12 
Input/output interface, 13 A MIDI device, 14 Sound-source equipment, 15 
Sound system, 16 A communication link interface, 17 A communication 
network, 18 Bus, 21 A sequence information-display field, the menu button for 
22 musical-piece selection, 23 The menu button for sequence selection, 24 
The menu button for scenario selection, and 25 An image display field, 26 An 
event field, 27-30 Carbon button for control 



